Skip to content

Add IEWC (Improved Elastic Weight Consolidation) plugin and strategy - #1702

Open
DavideWiest wants to merge 3 commits into
ContinualAI:masterfrom
DavideWiest:iewc
Open

DavideWiest wants to merge 3 commits into
ContinualAI:masterfrom
DavideWiest:iewc

Conversation

@DavideWiest

@DavideWiest DavideWiest commented Nov 16, 2026 •

Copy link
Copy Markdown

This adds IEWC as a new regularization method: IEWCPlugin in avalanche/training/plugins/iewc.py (subclass of EWCPlugin, same penalty and online/separate modes) and an IEWC strategy wrapper next to EWC.

EWC's empirical Fisher weights each old sample by the scale of its loss gradient, so samples the model already fits well barely count. IEWC divides each per-sample term by the squared norm of the loss gradient w.r.t. the model output (plus a small damping relative to the median), which removes that scale for any differentiable loss. EWC-DR fixes a related problem by reversing logits, which only applies to classification. On our held-out ImageNet-R evaluation this change alone gives +2.00 points over EWC.

Paper: https://arxiv.org/abs/2611.16039

The importance uses per-sample gradients, so it is slower than EWC's batched estimate. ewc_lambda follows Avalanche's EWC convention (no 1/2). Sample weights can be uniform, residual-diversity weighting (sample_weighting="rdw") or a custom function, and loss_scale_fn sets the output geometry.

Tests: new plugin tests (reduction to EWCPlugin with batch size 1, the damping formula, invariance to loss rescaling, all-zero case, RDW weights) and a strategy test like test_ewc. I also ran short Permuted MNIST runs next to EWC to check that it trains normally, and added examples/iewc.py. Docs lists are updated.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant